{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer learning with TensorFlow/Keras 2.0 \n",
    "\n",
    "In this example we'll implement fine-tuning and feature extracting transfer learning using the CIFAR-10 dataset. \n",
    "\n",
    "_This example is partially based on_ [https://github.com/tensorflow/docs/blob/master/site/en/tutorials/images/transfer_learning.ipynb]\n",
    "(https://github.com/tensorflow/docs/blob/master/site/en/tutorials/images/transfer_learning.ipynb)<br/>\n",
    "_The licensing information and the author of the base version are:<br/>\n",
    "License: Apache License, Version 2.0<br/>\n",
    "Copyright 2018 The TensorFlow Authors. All rights reserved.<br/>\n",
    "License: MIT<br/>\n",
    "Copyright 2017 François Chollet. All rights reserved.<br/>\n",
    "Copyright 2019 Ivan Vasilev.<br/>_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start with the imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll define the input image and batch size as constants:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMG_SIZE = 224\n",
    "BATCH_SIZE = 50"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll continue by loading the CIFAR-10 dataset using the `tensorflow_datasets` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, metadata = tfds.load('cifar10', with_info=True, as_supervised=True)\n",
    "\n",
    "raw_train, raw_test = data['train'].repeat(), data['test'].repeat()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define the input transformations (per sample) for the training and validation phases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_format_sample(image, label):\n",
    "    \"\"\"Transform data for training\"\"\"\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))\n",
    "    image = (image / 127.5) - 1\n",
    "    image = tf.image.random_flip_left_right(image)\n",
    "    image = tf.image.random_flip_up_down(image)\n",
    "\n",
    "    label = tf.one_hot(label, metadata.features['label'].num_classes)\n",
    "\n",
    "    return image, label\n",
    "\n",
    "\n",
    "def test_format_sample(image, label):\n",
    "    \"\"\"Transform data for testing\"\"\"\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))\n",
    "    image = (image / 127.5) - 1\n",
    "\n",
    "    label = tf.one_hot(label, metadata.features['label'].num_classes)\n",
    "\n",
    "    return image, label"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, we'll define the training and validation data providers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# assign transformers to raw data\n",
    "train_data = raw_train.map(train_format_sample)\n",
    "test_data = raw_test.map(test_format_sample)\n",
    "\n",
    "# extract batches from the training set\n",
    "train_batches = train_data.shuffle(1000).batch(BATCH_SIZE)\n",
    "test_batches = test_data.batch(BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll define 2 functions that build transfer learning models for either feature extacting, or fine-tuning. Both models use the `tf.keras.applications.ResNet50V2` ImageNet pretrained model. We'll start with feature extracting, which \"locks\" all model parameters (weights) except for the final fully-connected layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_fe_model():\n",
    "    \"\"\"\"Create feature extraction model from the pre-trained model ResNet50V2\"\"\"\n",
    "\n",
    "    # create the pre-trained part of the network, excluding FC layers\n",
    "    base_model = tf.keras.applications.ResNet50V2(input_shape=(IMG_SIZE, IMG_SIZE, 3),\n",
    "                                                  include_top=False,\n",
    "                                                  weights='imagenet')\n",
    "\n",
    "    # exclude all model layers from training\n",
    "    base_model.trainable = False\n",
    "\n",
    "    # create new model as a combination of the pre-trained net\n",
    "    # and one fully connected layer at the top\n",
    "    return tf.keras.Sequential([\n",
    "        base_model,\n",
    "        tf.keras.layers.GlobalAveragePooling2D(),\n",
    "        tf.keras.layers.Dense(\n",
    "            metadata.features['label'].num_classes,\n",
    "            activation='softmax')\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll continue with the fine-tuning model, which locks the first `fine_tune_at` layers, but trains all other model parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_ft_model():\n",
    "    \"\"\"\"Create fine tuning model from the pre-trained model ResNet50V2\"\"\"\n",
    "\n",
    "    # create the pre-trained part of the network, excluding FC layers\n",
    "    base_model = tf.keras.applications.ResNet50V2(input_shape=(IMG_SIZE, IMG_SIZE, 3),\n",
    "                                                  include_top=False,\n",
    "                                                  weights='imagenet')\n",
    "\n",
    "    # Fine tune from this layer onwards\n",
    "    fine_tune_at = 100\n",
    "\n",
    "    # Freeze all the layers before the `fine_tune_at` layer\n",
    "    for layer in base_model.layers[:fine_tune_at]:\n",
    "        layer.trainable = False\n",
    "\n",
    "    # create new model as a combination of the pre-trained net\n",
    "    # and one fully connected layer at the top\n",
    "    return tf.keras.Sequential([\n",
    "        base_model,\n",
    "        tf.keras.layers.GlobalAveragePooling2D(),\n",
    "        tf.keras.layers.Dense(\n",
    "            metadata.features['label'].num_classes,\n",
    "            activation='softmax')\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the `train_model` function, which builds takes the pre-built model, fits it over the training data, and plots the training and validation results. The function is shared for both feature extraction and fine-tuning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, epochs=5):\n",
    "    \"\"\"Train the model. This function is shared for both FE and FT modes\"\"\"\n",
    "\n",
    "    # configure the model for training\n",
    "    model.compile(optimizer=tf.keras.optimizers.Adam(lr=0.0001),\n",
    "                  loss='categorical_crossentropy',\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "    # train the model\n",
    "    history = model.fit(train_batches,\n",
    "                        epochs=epochs,\n",
    "                        steps_per_epoch=metadata.splits['train'].num_examples // BATCH_SIZE,\n",
    "                        validation_data=test_batches,\n",
    "                        validation_steps=metadata.splits['test'].num_examples // BATCH_SIZE,\n",
    "                        workers=4)\n",
    "\n",
    "    # plot accuracy\n",
    "    test_acc = history.history['val_accuracy']\n",
    "\n",
    "    plt.figure()\n",
    "    plt.plot(test_acc)\n",
    "    plt.xticks(\n",
    "        [i for i in range(0, len(test_acc))],\n",
    "        [i + 1 for i in range(0, len(test_acc))])\n",
    "    plt.ylabel('Accuracy')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now run the training procedure. Let's start by building the feature extraction model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "resnet50v2 (Model)           (None, 7, 7, 2048)        23564800  \n",
      "_________________________________________________________________\n",
      "global_average_pooling2d (Gl (None, 2048)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 10)                20490     \n",
      "=================================================================\n",
      "Total params: 23,585,290\n",
      "Trainable params: 20,490\n",
      "Non-trainable params: 23,564,800\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = build_fe_model()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's run the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 1000 steps, validate for 200 steps\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 226s 226ms/step - loss: 1.3464 - accuracy: 0.5466 - val_loss: 0.8496 - val_accuracy: 0.7100\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 215s 215ms/step - loss: 0.9247 - accuracy: 0.6885 - val_loss: 0.7822 - val_accuracy: 0.7464\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 224s 224ms/step - loss: 0.8354 - accuracy: 0.7167 - val_loss: 0.7722 - val_accuracy: 0.7605\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 226s 226ms/step - loss: 0.7826 - accuracy: 0.7349 - val_loss: 0.7767 - val_accuracy: 0.7680\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 222s 222ms/step - loss: 0.7582 - accuracy: 0.7392 - val_loss: 0.7683 - val_accuracy: 0.7751\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhV5bn38e9NIMxzEkQgBCGQqIxuseKAglbUVlutGGtbO5xjbau12Npqh9O3tnbwWD2e1npejrWtbd8iUm1RWydQnFAJCiIJhIAIQSQJECAMGe/3j71jt2EDO5CVtZP8PteVi6xnrbXzcyP7zvOs9azH3B0REZHmuoQdQEREUpMKhIiIJKQCISIiCalAiIhIQioQIiKSUNewA7SWjIwMz8nJCTuGiEi7snz58kp3z0y0r8MUiJycHAoLC8OOISLSrpjZu4fapyEmERFJSAVCREQSUoEQEZGEVCBERCQhFQgREUlIBUJERBJSgRARkYQ6zDwIEZHOZuuu/SxZW0GDO1efNrLVX18FQkSknaitb6Rw4w6WlFTw/NoK1m7bA8Dk7AEqECIinU3Zzn08vzZaEJaur2RvbQPd0oxTcwZx65Q8zhmXxdghfQL52SoQIiIp5EBdA6+/09RLKGd9xV4Ahg3oyScmD+OccVmcPnowfboH//GtAiEiErKNlXs/KAhLN2znQF0j6V27cNqoQVw1NZtzxmUxOrM3ZtamuVQgRETa2P7aBl7dsJ3n15azpKSCjdv3AZAzuBdXRkZwzrgsTjthEL3Sw/2IVoEQEQmYu7O+4l+9hNfe2UFtfSM9unXh9BMG84UzRjF9bCY5Gb3DjvohKhAiIgHYW1PPK+v/1Uso27kfgNGZvfnsR0YyfWwmU0cNoke3tJCTHpoKhIhIK3B3SrZVf1AQlm3cQV2D0ys9jWmjM7hu+mimj81kxKBeYUdNmgqEiMhR2n2gjpfXVbKkpIIlJRVs3XUAgHFD+vLF2LBRJGcQ6V3b50MrVCBERJLk7hRt3c3zaytYsraC5Zt20tDo9O3elTNzM7hxZibTx2UytH/PsKO2ChUIEZHDqNpXy4txvYSKPTUAnHR8P7589gmcMy6LydkD6JbWPnsJh6MCISISp7HRWbVl1wd3HK3YXEWjQ/+e3TgrN4PpYzOZPjaTrH49wo4aOBUIEen0tlfX8OK6Sp5fW84L6yrZsbcWM5gwrD/XnzuG6eOymDi8P107YC/hcAItEGY2C7gHSAPud/efN9t/N3BubLMXkOXuA8zsXODuuEPzgAJ3/1uQeUWkc2hodFZsrmJJ7I6jt7bswh0G9U7n7NwMzhmXxVm5GQzu0z3sqKEKrECYWRpwL3A+UAYsM7OF7l7UdIy7z4k7/gZgcqz9OWBSrH0QUAo8HVRWEen4yvcc4IWSaC/hxXWV7NpfRxeDSSMGMOe8sUwfm8n4Yf3p0qVtH2eRyoLsQUwFSt19A4CZzQMuBYoOcfxVwA8TtH8K+Ke77wskpYh0SHUNjby5qeqDeQmr39sNQEaf7pyXP4RzxmVyVm4GA3qlh5w0dQVZIIYBm+O2y4DTEh1oZiOBUcDiBLsLgLsOcd61wLUA2dnZx5JVRDqApgV0lpRU8NK6SvbU1JPWxTgleyA3XzCO6WMzOXFoP/USkpQqF6kLgAXu3hDfaGZDgfHAU4lOcve5wFyASCTiQYcUkdRyqAV0juvXg4snDGX62EzOyM2gX49uISdtn4IsEFuAEXHbw2NtiRQAX0vQPht41N3rWjmbiLRTyS6g09aPxu6IgiwQy4BcMxtFtDAUAJ9ufpCZ5QEDgaUJXuMq4NYAM4pIikulBXQ6m8DeUXevN7PriQ4PpQEPuPtqM7sNKHT3hbFDC4B57v6hISIzyyHaA1kSVEYRSU2puoBOZ2PNPpfbrUgk4oWFhWHHEJGjUNfQyLKNO1hUXM7iNeW8UxntJeQM7sX0sZkps4BOR2Rmy909kmif3m0RCcXOvbUsKang2eJtLCmpYM+BetLTuvCR0YO55vSRnDMuK+UW0OlsVCBEpE1EV1WrZlFxOYuKyyl8dweNDhl90rnw5OOYkTeEs3Iz6K1rCSlDfxMiEpja+n8NHS1as413Y2sv5w/tx9fOHcOMvCwmDh+geQkpSgVCRFrVzr21PLc22kt4oaSCPTX1pHftwrTRg/m3s05gZl4Wxw/oGOsldHQqECJyTNyddeVNQ0fbeGPTThodMvt256LxQ5mZn8WZuRm6wNwO6W9MRFqstr6R197Z/sHQ0eYd+4HoIjrXz8hlZl6WHnzXAahAiEhStlfX8NzaChav2cYLJZVU19TTvWsXzhiTwXXTRzMjL6vDLLUpUSoQIpKQu1OyrZpni7exqHgbb26uwh2y+nbn4xOHMjNvCGeMyaBnelrYUSUgKhAi8oGa+gZe3bCDxcXbWLSmnLKd0aGj8cP6c+PMXGbmDeGk4/U01M5CBUKkk6usrmHxmnIWF5fz4roK9tY20KNbF84ck/HBrahDOsH6y3IwFQiRTsbdWfP+HhYVb+PZ4nJWlkWHjo7r14NLJw/jvPwspo3OoEc3DR11dioQIp3AgboGXt2w/YNnHW2pig4dTRzen2/MHMvM/CxOOr6fHn4nH6ICIdJBle85wHNrohPWXiqtZF9tAz27pXFmbgY3zIgOHWVp6EgOQwVCpINwd4q27o7NTShn5eYqAI7v34PLpgxjZv4QTj9hsIaOJGkqECLt2IG6Bpau386zxdtYvKacrbsOYAYThw/gm+ePZWb+EPKH9tXQkRwVFQiRdqZ89wEWrynn2eJyXi6tZH9dA73S0zgrN4M5543l3LwsMvt2DzumdAAqECIpzt1Z/d7uD3oJb5XtAqJLbl4RGc7M/CGcNmqQho6k1alAiKSg/bUNvLK+kmeLy1m8ZhvbdtdgBpNHDODmC8YxMz+LcUM0dCTBUoEQSRHv74oOHS0q3sbL6ys5UNdI7/Q0zh6byYy8LM7NyyKjj4aOpO2oQIiEpLHRefu9XR88EfXtLbsBGD6wJwWnZjMzP4upowbRvauGjiQcKhAibWh/bQMvlVayeM02FhWXU74nOnQ0JXsg3541jvPyh5Cb1UdDR5ISAi0QZjYLuAdIA+5395832383cG5ssxeQ5e4DYvuygfuBEYADF7n7xiDzigShfPcBni6KPhH1lfXbqalvpE/3rkyPGzoa1Ds97JgiBwmsQJhZGnAvcD5QBiwzs4XuXtR0jLvPiTv+BmBy3Es8CNzu7s+YWR+gMaisIkHYc6COe59bzwMvv0NtfSPZg3rx6dOymZk3hKmjBpHetUvYEUUOK8gexFSg1N03AJjZPOBSoOgQx18F/DB27IlAV3d/BsDdqwPMKdKq6hsaeahwM3c9XcL2vbVcNmUY100fraEjaXeCLBDDgM1x22XAaYkONLORwChgcaxpLFBlZo/E2p8FbnH3hmbnXQtcC5Cdnd2q4UWOxpKSCm5/ooiSbdVMHTWI312cz4ThA8KOJXJUUuUidQGwIK4AdAXOIjrktAl4CPg88Nv4k9x9LjAXIBKJeFuFFWmuZNsebn+imCUlFYwc3Iv/+cwpXHDSEPUYpF0LskBsIXqBucnwWFsiBcDX4rbLgBVxw1N/Az5CswIhErbK6hrufqaEv7y+iT7du/L9i/P53Ok5ur4gHUKQBWIZkGtmo4gWhgLg080PMrM8YCCwtNm5A8ws090rgBlAYYBZRVrkQF0Dv3t5I/c+V8qBugY+d3oON87MZaDuRpIOJLAC4e71ZnY98BTR21wfcPfVZnYbUOjuC2OHFgDz3N3jzm0ws28BiyzaR18O/G9QWUWS5e48/tZWfv7PNWyp2s95+UO49aI8Rmf2CTuaSKuzuM/ldi0SiXhhoToZEpw3Nu3kx48X8eamKvKH9uMHF+czbUxG2LFEjomZLXf3SKJ9qXKRWiRlbd6xjzueWstjK98js2937vjUBC6fMpy0LroALR2bCoTIIew5UMdvnl/Pb196hy4GX5+Zy5fPPoHe3fXPRjoH/Z8u0sxBE90mD+NbF4zj+AE9w44m0qZUIETifGiiW84gfvcFTXSTzksFQoREE92mcMFJx2mim3RqKhDSqcVPdOsdm+j22dNHag0GEVQgpJOKn+i2PzbR7eszc/XYbZE4KhDSqRw80S2LWy7MZ0yWJrqJNKcCIZ3GG5t28pPHi3gjNtHtjk9N4AxNdBM5JBUI6fDKdu7jjifXsrBpotvlE7j8FE10EzkSFQjpsPYcqOO+59dzf9NEtxlj+PL00ZroJpIk/UuRDqe+oZH5hWXc9cxaKqs10U3kaKlASIfyQkkFtz9RzNptezg1ZyAPfP5UTXQTOUoqENIhrNu2h9v/UczzayvIHtSL+66ewqyTNdFN5FioQEi7Vlldw389W8JfXt9Mr/Q0vndRPp+bpoluIq1BBULapQN1Dfz+lY3cu7iUfXUNfOa0bG48b6wmuom0IhUIaVfcnSdWRSe6le3cz8y8LG69SBPdRIKgAiHtxpuxFd3e2FRF3nF9+dOXTuPMXE10EwmKCoSkvPiJbhl9uvOLy8fzqVNGaKKbSMBUICRlxU90M+CG2ES3PproJtIm9C9NUk7ziW6fnDyMmzXRTaTNBVogzGwWcA+QBtzv7j9vtv9u4NzYZi8gy90HxPY1AKti+za5+yVBZpXU0Hyi22+vOZWJIzTRTSQMgRUIM0sD7gXOB8qAZWa20N2Lmo5x9zlxx98ATI57if3uPimofJJaNNFNJPUE2YOYCpS6+wYAM5sHXAoUHeL4q4AfBphHUtD26hru1kQ3kZQUZIEYBmyO2y4DTkt0oJmNBEYBi+Oae5hZIVAP/Nzd/5bgvGuBawGys7NbKba0BU10E0l9qXKRugBY4O4NcW0j3X2LmZ0ALDazVe6+Pv4kd58LzAWIRCLednHlaLk7/1j1Pj9/spjNO/YzIy+L716Ux5isvmFHE5FmgiwQW4ARcdvDY22JFABfi29w9y2xPzeY2fNEr0+sP/hUaS9WbK7ix48XsfzdneQd15c/fmkqZ+Vmhh1LRA7hiAUidvH4T+6+s4WvvQzINbNRRAtDAfDpBK+fBwwElsa1DQT2uXuNmWUAZwB3tPDnS4rYUrWfO55cw99XRCe6/fyy8VwR0UQ3kVSXTA9iCNE7kN4AHgCecvcjDue4e72ZXQ88RfQ21wfcfbWZ3QYUuvvC2KEFwLxmr5kP/F8zawS6EL0GcaiL25Kiqmvque/5Uu5/8R0Arj93DNedo4luIu2FJfFZj0XvNfwo8AUgAswHftv8mkCYIpGIFxYWhh1DgIZGZ37hZn75dAmV1TV8YtLx3Dwrj2Ga6CaScsxsubtHEu1L6lc5d3czex94n+hdRQOBBWb2jLt/u/WiSnv34rroRLc17+8hMnIg918TYZImuom0S8lcg7gR+BxQCdwP3OzudWbWBVgHqEAIpeV7uP2JYp5bW8GIQT35zdVTuFAT3UTatWR6EIOAy9z93fhGd280s48FE0vai+3VNdyzaB1/fm0Tvbql8d2L8rhmWo4muol0AMkUiH8CO5o2zKwfkO/ur7l7cWDJJKXV1Dfwh1c28qvFpeyrbeDq07K5cWYug/t0DzuaiLSSZArEfcCUuO3qBG3SiTxTtI3bHl+tiW4iHVwyBcLib0GNDS3pPsVO6rUN27n2j4WMG6KJbiIdXTIf9BvM7OtEew0AXwU2BBdJUlV1TT3ffHgl2YN68devTKO35jOIdGhdkjjmOmAa0dnQTQ/cuzbIUJKafvJ4Ee9V7eeu2RNVHEQ6gSP+K3f3cqKznaUTe7ZoG/OWbear54zmlJGDwo4jIm0gmXkQPYAvAScBPZra3f2LAeaSFLK9uoZbHnmL/KH9+MZ5Y8OOIyJtJJkhpj8CxwEXAEuIPpV1T5ChJHW4O9979G1276/nrtkTSe+azP8yItIRJPOvfYy7/wDY6+5/AC7mEAv/SMfztxVbeHL1+9z00bHkD+0XdhwRaUPJFIi62J9VZnYy0B/ICi6SpIr3qvbzH39fzak5A/n3s04IO46ItLFkbkWZG1uf4fvAQqAP8INAU0noGhudmxespKHR+eUVk7R2g0gndNgCEXsg3+7YYkEvAPo1spN4cOlGXi7dzs8uG0/24F5hxxGREBx2iMndG9HTWjud0vJqfvbPNZw7LpOCU0cc+QQR6ZCSuQbxrJl9y8xGmNmgpq/Ak0ko6hoa+eb8FfRKT+MXl0/Q47pFOrFkrkFcGfvza3FtjoabOqTfPLeelWW7+M3VU8jq1+PIJ4hIh5XMTOpRbRFEwvdWWRW/WryOT0w6novGDw07joiELJmZ1J9L1O7uD7Z+HAnLgboGbpq/kow+3fnRJSeHHUdEUkAyQ0ynxn3fA5gJvAGoQHQgdzy5ltLyav74pan079Ut7DgikgKSGWK6IX7bzAYA8wJLJG3ulfWVPPDyO1xz+kit7yAiHziaB+vsBZK6LmFms8xsrZmVmtktCfbfbWYrYl8lZlbVbH8/Myszs18fRU5Jwu4Dddz88FuckNGbWy7MDzuOiKSQZK5BPEb0riWIFpQTgflJnJcG3AucT3QdiWVmttDdi5qOcfc5ccffAExu9jI/JjpBTwJy22NFbN21n79+ZRo909PCjiMiKSSZaxB3xn1fD7zr7mVJnDcVKHX3DQBmNg+4FCg6xPFXAT9s2jCzU4AhwJNAJImfJy301Or3WbC8jBtmjGFy9sCw44hIikmmQGwCtrr7AQAz62lmOe6+8QjnDQM2x203rUZ3EDMbSXTYanFsuwvwS+AzwHmH+gFmdi2x1e2ys7OT+E+RJpXVNXz3kVWcPKwfN8zIDTuOiKSgZK5BPAw0xm03xNpaUwGwwN0bYttfBf5xpJ6Ku89194i7RzIzdXE1We7OrY+sYk9NPXfNnqQ1HkQkoWR6EF3dvbZpw91rzSw9ifO2APEP8hkea0ukgA/P1D4dOMvMvkr06bHpZlbt7gdd6JaWW7C8jGeKtvH9i/MZO6Rv2HFEJEUlUyAqzOwSd18IYGaXApVJnLcMyDWzUUQLQwHw6eYHmVkeMBBY2tTm7lfH7f88EFFxaB1lO/fxo8eKOG3UIL54hibJi8ihJVMgrgP+HHeraRmQcHZ1PHevN7PrgaeANOABd19tZrcBhU0Fh2jhmOfufqjXktbR2Oh86+GVANx5xUS6aI0HETkMS/Zz2cz6ALh7daCJjlIkEvHCwsKwY6S0+1/cwE+eKOaOT01gdkSP8RYRMLPl7p7wTtEjXp00s5+a2QB3r3b3ajMbaGY/af2YEqR12/Zwx1NrOS9/CFecMjzsOCLSDiRz+8qF7v7BDOfY6nIXBRdJWltdQyNz5q+gT/eu/Oyy8VrjQUSSkkyBSDOz7k0bZtYT6H6Y4yXF/GrROt7espuffnI8mX31VyciyUnmIvWfgUVm9jvAgM8DfwgylLSeNzft5N7n13P5lOHMOvm4sOOISDuSzNNcf2FmK4nOaHaidyWNDDqYHLv9tQ18c/5KhvTtzg8vOTHsOCLSziQ7hXYb0eJwBTADKA4skbSaXzy5hg2Ve7nzion066E1HkSkZQ7ZgzCzsUQfoHcV0YlxDxG9LfbcNsomx+DFdRX8/pWNfOGMHKaNyQg7joi0Q4cbYloDvAh8zN1LAcxszmGOlxSxa190jYfRmb35zqy8sOOISDt1uCGmy4CtwHNm9r9mNpPoRWpJcf/nsdVUVNdw95WT6NFNazyIyNE5ZIFw97+5ewGQBzwHfAPIMrP7zOyjbRVQWuYfq7by6JtbuGHGGCYMHxB2HBFpx454kdrd97r7/3P3jxN9IuubwHcCTyYtVr77AN97dBUThvfna+eOCTuOiLRzLVoIwN13xtZgmBlUIDk67s4tj6xiX20Dd82eRLc0rfEgIsdGnyIdxEPLNrN4TTm3XJjHmKw+YccRkQ5ABaID2LR9Hz9+vIhpowdzzek5YccRkQ5CBaKda4it8dDFjP/UGg8i0oqSeRaTpLD7X9zA6xt3cNfsiQwb0DPsOCLSgagH0Y6teX83v3y6hFknHccnJw8LO46IdDAqEO1UbX0jcx5aSb+eXbn9kydrjQcRaXUaYmqn7llUQvHW3dz/uQiD+2iNBxFpfepBtEPL393Bfc+v58rICM47cUjYcUSkg1KBaGf21tRz0/yVHD+gJ9//WH7YcUSkAwu0QJjZLDNba2alZnZLgv13m9mK2FeJmVXF2kea2Rux9tVmdl2QOduTn/2zmE079nHnFRPpqzUeRCRAgV2DMLM04F7gfKAMWGZmC929qOkYd58Td/wNwOTY5lbgdHevMbM+wNuxc98LKm97sKSkgj+9uol/P2sUHzlhcNhxRKSDC7IHMRUodfcN7l4LzAMuPczxVwF/AXD3WnevibV3Dzhnu1C1r5abH17J2CF9+OZHx4UdR0Q6gSA/eIcBm+O2y2JtBzGzkcAoYHFc2wgzeyv2Gr9I1Hsws2vNrNDMCisqKlo1fKr5wd9Xs2NvLXfN1hoPItI2UuU38wJggbs3NDW4+2Z3nwCMAa4xs4Nu14k9WTbi7pHMzMw2jNu2Fq58j8dWvsc3zsvl5GH9w44jIp1EkAViCzAibnt4rC2RAmLDS83Feg5vA2e1arp2YtvuA/zgb28zacQArps+Ouw4ItKJBFkglgG5ZjbKzNKJFoGFzQ8yszxgILA0rm24mfWMfT8QOBNYG2DWlOTu3LzgLWrqG7hr9kS6ao0HEWlDgd3F5O71ZnY98BSQBjzg7qvN7Dag0N2bikUBMM/dPe70fOCXZuZE18G+091XBZU1Vf35tU28UFLBjy89iRMytcaDiLQt+/DncvsViUS8sLAw7BitZmPlXi6850UiOQN58ItT9awlEQmEmS1390iifRqzSEENjc5N81fQLc2441MTVBxEJBR6WF8K+p8l63ljUxX3FExiaH+t8SAi4VAPIsWsfm8X//VsCRdPGMolE48PO46IdGIqECmkpr6Bmx5ayYBe6fzkUq3xICLh0hBTCrnrmRLWbtvD7z5/KgN7p4cdR0Q6OfUgUsTr7+xg7gsbuGpqNufmZYUdR0REBSIVVNfU882HVzBiYC++f7HWeBCR1KAhphRw+xNFlO3cz8NfPp3e3fVXIiKpQT2IkC1es42/vL6ZL589mkjOoLDjiIh8QAUiRDv21vLtBavIO64vc87PDTuOiMiHaDwjJO7O9/+2il37a3nwi1Pp3lVrPIhIalEPIiR/X/Ee/1j1PjedP44Tj+8XdhwRkYOoQIRg6679/ODvb3PKyIFce/YJYccREUlIBaKNNTY6Nz/8Fg2Nzl2zJ5LWRbOlRSQ1qUC0sT+++i4vlVbyvYvzGTm4d9hxREQOSQWiDa2vqOZn/yzmnHGZfHpqdthxREQOSwWijdQ3NHLT/JX06JbGHZdrjQcRSX26zbWN3Pf8elZuruLXn55MVr8eYccRETki9SDawKqyXdyzaB2XTDyej03QGg8i0j6oQATsQF0Dc+avYHCfdH586clhxxERSZqGmAJ251NrKS2v5sEvTqV/r25hxxERSVqgPQgzm2Vma82s1MxuSbD/bjNbEfsqMbOqWPskM1tqZqvN7C0zuzLInEFZun47v335HT77kZGcPTYz7DgiIi0SWA/CzNKAe4HzgTJgmZktdPeipmPcfU7c8TcAk2Ob+4DPufs6MzseWG5mT7l7VVB5W9ueA3V86+GV5Azuza0X5YUdR0SkxYLsQUwFSt19g7vXAvOASw9z/FXAXwDcvcTd18W+fw8oB9rVr+C3PVbE1l37+eXsifRK10ieiLQ/QRaIYcDmuO2yWNtBzGwkMApYnGDfVCAdWB9AxkA8vfp9Hl5exlfPGcOU7IFhxxEROSqpchdTAbDA3RviG81sKPBH4Avu3tj8JDO71swKzaywoqKijaIeXmV1Dbc+sooTh/bj6zO1xoOItF9BFogtwIi47eGxtkQKiA0vNTGzfsATwPfc/dVEJ7n7XHePuHskMzP8ESh357uPrGLPgXruvnIS6V1Tpf6KiLRckJ9gy4BcMxtlZulEi8DC5geZWR4wEFga15YOPAo86O4LAszYqv76xhaeLtrGzReMY9xxfcOOIyJyTAIrEO5eD1wPPAUUA/PdfbWZ3WZml8QdWgDMc3ePa5sNnA18Pu422ElBZW0NZTv38aOFq5k6ahBfPHNU2HFERI6Zffhzuf2KRCJeWFgYys9ubHSuvv813iqr4slvnM2IQb1CySEi0lJmttzdI4n2aZC8Ffz+lY0s3bCd//j4iSoOItJhqEAco9LyPfziyTWcl5/F7MiII58gItJOqEAcg7qGRuY8tJLe3bvys8u0xoOIdCya4nsMfr24lFVbdnHf1VPI7Ns97DgiIq1KPYijtHJzFb9+rpTLJg/jwvFDw44jItLqVCCOwv7a6BoPWX2788NLTgo7johIIDTEdBR+8eQaNlTs5c//dhr9e2qNBxHpmNSDaKGXSyv5/Ssb+fy0HM4YkxF2HBGRwKhAtMCu/dE1Hk7I7M13ZmmNBxHp2DTE1AI/Wria8j01PPKVafRMTws7johIoNSDSNI/V23lkTe3cP25Y5g4YkDYcUREAqcCkYTyPQf47qOrGD+sP9fPGBN2HBGRNqECcQRNazzsrW3g7isn0i1Nb5mIdA76tDuC+YWbeba4nO/MymNMltZ4EJHOQwXiMDbv2MdtjxVx+gmD+cK0nLDjiIi0KRWIQ2hodL45fyVdzLhz9kS6dNGD+ESkc1GBOIQHXnqH1zfu4IeXnMSwAT3DjiMi0uZUIBJY+/4e/vOptXz0xCFcPmVY2HFEREKhAtFMbX0jcx5aQb+eXfnpZeO1xoOIdFqaSd3Mfy9aR9HW3cz97Clk9NEaDyLSeakHEeeNTTv5zfOlXHHKcD560nFhxxERCZUKRMy+2npuemgFQ/v35D8+fmLYcUREQhdogTCzWWa21sxKzeyWBPvvNrMVsa8SM6uK2/ekmVWZ2eNBZmzys3+s4d0d+7jzion07aE1HkREArsGYWZpwL3A+UAZsMzMFrp7UdMx7j4n7vgbgMlxL/GfQC/gy0FlbLKkpII/vmrNlvIAAAVFSURBVPou/3bmKE4fPTjoHyci0i4E2YOYCpS6+wZ3rwXmAZce5virgL80bbj7ImBPgPkA2LWvjm8vWEluVh++dcG4oH+ciEi7EWSBGAZsjtsui7UdxMxGAqOAxS35AWZ2rZkVmllhRUXFUYWsa2xk/LAB3DV7Ej26aY0HEZEmqXKbawGwwN0bWnKSu88F5gJEIhE/mh+c0ac7918TOZpTRUQ6tCB7EFuAEXHbw2NtiRQQN7wkIiLhC7JALANyzWyUmaUTLQILmx9kZnnAQGBpgFlERKSFAisQ7l4PXA88BRQD8919tZndZmaXxB1aAMxz9w8NEZnZi8DDwEwzKzOzC4LKKiIiB7Nmn8vtViQS8cLCwrBjiIi0K2a23N0TXojVTGoREUlIBUJERBJSgRARkYRUIEREJKEOc5HazCqAd4/hJTKAylaK0xno/WoZvV8to/erZY7l/Rrp7pmJdnSYAnGszKzwUFfy5WB6v1pG71fL6P1qmaDeLw0xiYhIQioQIiKSkArEv8wNO0A7o/erZfR+tYzer5YJ5P3SNQgREUlIPQgREUlIBUJERBLq9AXCzB4ws3IzezvsLKnOzEaY2XNmVmRmq83sxrAzpTIz62Fmr5vZytj79aOwM7UHZpZmZm+a2eNhZ2kPzGyjma0ysxVm1qpPLO301yDM7GygGnjQ3U8OO08qM7OhwFB3f8PM+gLLgU+4e1HI0VKSmRnQ292rzawb8BJwo7u/GnK0lGZmNwERoJ+7fyzsPKnOzDYCEXdv9YmFnb4H4e4vADvCztEeuPtWd38j9v0eout8JFxnXMCjqmOb3WJfnfs3siMws+HAxcD9YWcRFQg5SmaWA0wGXgs3SWqLDZesAMqBZ9xd79fh/RfwbaAx7CDtiANPm9lyM7u2NV9YBUJazMz6AH8FvuHuu8POk8rcvcHdJxFdk32qmWkY8xDM7GNAubsvDztLO3Omu08BLgS+Fhs2bxUqENIisbH0vwJ/dvdHws7TXrh7FfAcMCvsLCnsDOCS2Jj6PGCGmf0p3Eipz923xP4sBx4FprbWa6tASNJiF11/CxS7+11h50l1ZpZpZgNi3/cEzgfWhJsqdbn7re4+3N1ziK5Vv9jdPxNyrJRmZr1jN4xgZr2BjwKtdkdmpy8QZvYXYCkwzszKzOxLYWdKYWcAnyX6m92K2NdFYYdKYUOB58zsLWAZ0WsQunVTWtMQ4CUzWwm8Djzh7k+21ot3+ttcRUQksU7fgxARkcRUIEREJCEVCBERSUgFQkREElKBEBGRhFQgRFrAzBribvFdYWa3tOJr5+ipwpJKuoYdQKSd2R97dIZIh6cehEgriD2T/47Yc/lfN7MxsfYcM1tsZm+Z2SIzy461DzGzR2NrRaw0s2mxl0ozs/+NrR/xdGwGtkgoVCBEWqZnsyGmK+P27XL38cCviT6VFOBXwB/cfQLwZ+C/Y+3/DSxx94nAFGB1rD0XuNfdTwKqgMsD/u8ROSTNpBZpATOrdvc+Cdo3AjPcfUPsgYbvu/tgM6skushSXax9q7tnmFkFMNzda+JeI4fo4zhyY9vfAbq5+0+C/y8TOZh6ECKtxw/xfUvUxH3fgK4TSohUIERaz5Vxfy6Nff8K0SeTAlwNvBj7fhHwFfhgUaH+bRVSJFn67USkZXrGVohr8qS7N93qOjD25NYa4KpY2w3A78zsZqAC+EKs/UZgbuzpwQ1Ei8XWwNOLtICuQYi0giAXjhcJi4aYREQkIfUgREQkIfUgREQkIRUIERFJSAVCREQSUoEQEZGEVCBERCSh/w9i9LzxELT0mgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "train_model(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also try the fine-tuning model for comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "resnet50v2 (Model)           (None, 7, 7, 2048)        23564800  \n",
      "_________________________________________________________________\n",
      "global_average_pooling2d_1 ( (None, 2048)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 10)                20490     \n",
      "=================================================================\n",
      "Total params: 23,585,290\n",
      "Trainable params: 20,580,362\n",
      "Non-trainable params: 3,004,928\n",
      "_________________________________________________________________\n",
      "Train for 1000 steps, validate for 200 steps\n",
      "Epoch 1/5\n",
      "1000/1000 [==============================] - 302s 302ms/step - loss: 0.4756 - accuracy: 0.8367 - val_loss: 0.4555 - val_accuracy: 0.8452\n",
      "Epoch 2/5\n",
      "1000/1000 [==============================] - 295s 295ms/step - loss: 0.2451 - accuracy: 0.9170 - val_loss: 0.4341 - val_accuracy: 0.8588\n",
      "Epoch 3/5\n",
      "1000/1000 [==============================] - 294s 294ms/step - loss: 0.1790 - accuracy: 0.9399 - val_loss: 0.3699 - val_accuracy: 0.8748\n",
      "Epoch 4/5\n",
      "1000/1000 [==============================] - 296s 296ms/step - loss: 0.1474 - accuracy: 0.9502 - val_loss: 0.3978 - val_accuracy: 0.8757\n",
      "Epoch 5/5\n",
      "1000/1000 [==============================] - 294s 294ms/step - loss: 0.1242 - accuracy: 0.9584 - val_loss: 0.3802 - val_accuracy: 0.8821\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxV5b3v8c+PJBBA5kkgxICCDKIM24DWoXVEasW2VkFEQAZtq1Vr6+Hco7deOtweO9xOHCrzIIocayvHqlhbqmgZksgggwOEKQGZ5yFk+N0/9sLuxiB7QzYrw/f9eu1X9nr2Wk9+e4v7m7We9axl7o6IiEi86oRdgIiIVC8KDhERSYiCQ0REEqLgEBGRhCg4REQkIalhF3AutGzZ0rOyssIuQ0SkWsnLy9vt7q3Kt9eK4MjKyiI3NzfsMkREqhUz21xRuw5ViYhIQhQcIiKSEAWHiIgkRMEhIiIJUXCIiEhCFBwiIpIQBYeIiCREwSEiUsO4Ozmb9vL06x8kpf9aMQFQRKQ2KCktY8GaHUxelM+KrftpUj+N4Vdm0aZxeqX+HgWHiEg1d6SohHm5W5n6zkYK9h3jghYNGD+oB3f0zaBB3cr/mldwiIhUU58cOM6Mf2ziuaWbOXi8hL4XNOOJL3fjxu7nk1LHkvZ7FRwiItXMuu0Hmbwon/9ZuY3SMufmHucz+upO9L2g2Tn5/QoOEZFqwN15++PdTFmUz6KPd1M/LYW7szO576qOXNCi4TmtRcEhIlKFFZWUMn/FNqa+s5EPPjlE60b1+P7NFzO0XyZNG9QNpSYFh4hIFbT/6AnmLN3CzH9sYuehIi5u04if3XEpt/VqR73UlFBrU3CIiFQhW/YcZdq7G3khZyvHiku5unNLfvaNy7imc0vMkjfgnYikBoeZDQB+DaQAU9z9p+VezwRmAk2Ddca5+6tmlgZMAfoENc5y9/8bT58iItVR3uZ9TFmUz4I1n5BSx/jKZe0YfVUnurdrHHZpn5G04DCzFGACcCNQAOSY2Xx3Xxuz2hPAPHefaGbdgVeBLOAbQD1372lmDYC1ZvY8sDWOPkVEqoXSMucvaz9h8qKN5G3eR+P0VO6/9kKGX5HF+U0qd9JeZUrmHkc2sN7d8wHMbC4wCIj9knfgZJw2AbbFtDc0s1SgPnACOBhnnyIiVdrREyW8mFfA1Hc2snnPUTo0r88PvtKdOyMdaFiv6o8gJLPC9kT3EE4qAPqVW+cp4A0zewhoCNwQtL9INBC2Aw2AR919r5nF0ycAZjYWGAuQmZl5Vm9ERKQy7Dx0nFn/2MyzSzez/2gxvTo05fGbu3JzjzakplSfSweGHW1DgBnu/gszuwKYbWaXEN2zKAXaAc2ARWb2ZiIdu/skYBJAJBLxyi1bRCR+H+04xOS383l5xTaKy8q4sVsbxl4TnbBXVQa8E5HM4CgEOsQsZwRtsUYBAwDcfbGZpQMtgbuB1929GNhpZu8CEaJ7G6frU0QkdO7Ou+v3MHlRPm99tIv0tDrcdXkH7ruqIx1bntsJe5UtmcGRA3Q2s45Ev9wHEw2EWFuA64EZZtYNSAd2Be3XEd0DaQj0B35FdCzjdH2KiITmREkZr6zaxuRFG1m3/SAtz6vHYzd2YWj/C2jeMJwJe5UtacHh7iVm9iCwgOips9PcfY2ZjQdy3X0+8Bgw2cweJTogPsLd3cwmANPNbA1gwHR3XwVQUZ/Jeg8iIvE6cKyY55dtYca7m/jk4HE6tz6P//x6Twb1ak96WrgT9iqbudf8w/+RSMRzc3PDLkNEaqCte48y/d1NvJCzhSMnSrnywhaMuaYT13ZuRZ0kXqH2XDCzPHePlG8Pe3BcRKRaWrF1P5MX5fPa+9upY9EJe6Ou6sgl7ZuEXVrSKThEROJUVua8uW4HUxZtZNmmvTSql8qYqzsx4gtZtG1SP+zyzhkFh4jIaRw7Ucof3itg2jsbyd99hPZN6/PEl7tx1+UdaJSeFnZ555yCQ0TkFHYfLmLW4s3MXryJfUeLuTSjCb8d0ptbLjm/Wk3Yq2wKDhGRctbvPMzUd/L5w3uFnCgp44ZubRhzdUeyOzavlhP2KpuCQ0SE6IS9Jfl7mbwon799sJN6qXW4o28Go67qyIWtzgu7vCpFwSEitVpxaRmvvr+dyYvyWV14kBYN6/LIDZ0Z1v8CWpxXL+zyqiQFh4jUSoeOFzN32Vamv7uRbQeO06lVQ37y1Z58rU/Nm7BX2RQcIlKrbNt/jOnvbmTusq0cKiqhf6fm/PD2S/jSxa2r/YS9c0XBISK1wvsFB5i8KJ8/v78dgC/3bMuYqzvRM6PmT9irbAoOEamxysqchR/uZPKifJbk7+W8eqmMvDKLkVd1pH3T2jNhr7IpOESkxjleXMoflxcyZVE+G3YdoW2TdP7XwK4Mzs6kcS2csFfZFBwiUmPsPXKC2Ys3M3vJJnYfPkGPdo359eBeDOzZlrRaPGGvsik4RKTay991mKnvbOTFvAKKSsq4rmtrRl/dkSs6tdCEvSRQcIhIteTu5Gzax6S38/nrBztIS6nD13q3Z/TVHbmodaOwy6vRFBwiUq2UlJbx+ppPmPx2PisLDtCsQRoPfekihl2RRatGmrB3Lig4RKTKcncOF5Vw8HgJB48V848Ne5j2zkYK9x+jY8uG/Oj2S/h6nwzq19WEvXNJwSEiSePuHC8u48CxYg4eL+Zg8PPAsWIOHiv5dPngsZJ/rhOzfOh4MWXlblKandWcH3ylOzd0a6MJeyFJanCY2QDg10TvDz7F3X9a7vVMYCbQNFhnnLu/amZDge/HrHop0MfdV5jZ34G2wLHgtZvcfWcy34dIbVZUUhr9kv/0C7/40z2AfwmBk8FQ7vXi0s+/PXX9tBSa1E+jcf1UGqen0bpROhe1Sg3a0mic/s/XLmjRkO7tGp+jdy6nkrTgMLMUYAJwI1AA5JjZfHdfG7PaE8A8d59oZt2BV4Esd58DzAn66Qn8yd1XxGw31N11E3GROJSUlv3LF/m//HV/yraST0OiqKTsc/uvm1In+gUffLk3bVCXzBYNaZyeSuP6adEAiPny/2dbKo3S06ibqtNkq5tk7nFkA+vdPR/AzOYCg4DY4HDg5J8PTYBtFfQzBJibxDpFqrSyMudQUfSL/0DMF/2p/sIvvwdw5ETp5/afUsc+/SI/+Rf++U3SY77wY14L2prEhIAuCFj7JDM42gNbY5YLgH7l1nkKeMPMHgIaAjdU0M9dRAMn1nQzKwX+APzI3T+zL2xmY4GxAJmZmWdSv0hSbdlzlL9+sKOCw0DF/9J2uKiEz/4L/yczaFQvNeZLPY2slg0+89d945ggiD001KBuiuY6SELCHhwfAsxw91+Y2RXAbDO7xN3LAMysH3DU3VfHbDPU3QvNrBHR4BgGzCrfsbtPAiYBRCKRzz/IKnKObdlzlK9NfJfdh08AcF691H/5cm/XNJ2u9Rv9y1/8FR3zb9IgjfPqpmqQWM6pZAZHIdAhZjkjaIs1ChgA4O6LzSwdaAmcHOweDDwfu4G7FwY/D5nZc0QPiX0mOESqqj2Hixg+fRklZc6fv3MVF7dpVKvvXy3VTzL/teYAnc2so5nVJRoC88utswW4HsDMugHpwK5guQ5wJzHjG2aWamYtg+dpwK3AakSqiWMnShk1M5dt+48x5d4IPdo1UWhItZO0PQ53LzGzB4EFRE+1nebua8xsPJDr7vOBx4DJZvYo0YHyETHjFdcAW08OrgfqAQuC0EgB3gQmJ+s9iFSmktIyHnp+OSsL9jNxaF8iWc3DLknkjFgF48o1TiQS8dxcnb0r4XF3/uNPq3lu6RbGD+rBvVdkhV2SyGmZWZ67R8q3ax9Z5ByYsHA9zy3dwgPXXqjQkGpPwSGSZC/mFfDzNz7i9l7tePzmi8MuR+SsKThEkujtj3Yx7g+r+MJFLXj6jst02qzUCAoOkSRZXXiAbz6bR+c2jfj9PX11aQ2pMfQvWSQJtu49yojpOTRtUJcZIy+nke5zLTVI2DPHRWqcfUdOMHz6Mk6UlPL8mH60aZwedkkilUrBIVKJjheXMmpmDgX7jvHsqH50bqNbmErNo0NVIpWktMz5zvPLWb51P7++qxfZHTXBT2omBYdIJXB3npq/hjfW7uB/39qdW3q2DbskkaRRcIhUgolvbWD2ks2MvaYTI7/QMexyRJJKwSFyll56r4CnX/+Q2y5rx7gBXcMuRyTpFBwiZ2HRx7t4/MVVXNGpBT/7xqWa4Ce1goJD5Ayt2XaAbz77Hhe1Po9n7u1LvVTdQlVqBwWHyBko2Bed4Nc4PZUZI7NprAl+UosoOEQStP/oCYZPW0ZRcSkz7svm/Caa4Ce1iyYAiiTgeHEpo2fmsnXvMWaNyqaLJvhJLaTgEIlTaZnzyNwV5G7ex+/u7k3/Ti3CLkkkFDpUJRIHd+eHr6zl9TWf8OSt3bn10nZhlyQSmqQGh5kNMLMPzWy9mY2r4PVMM1toZsvNbJWZDQzah5rZiphHmZn1Cl7ra2bvB33+xsx0/qMk3aS385nxj02Mvqojo67SBD+p3ZIWHGaWAkwAbgG6A0PMrHu51Z4A5rl7b2Aw8F8A7j7H3Xu5ey9gGLDR3VcE20wExgCdg8eAZL0HEYCXVxTyf1/7gFsvbcv/Gtgt7HJEQpfMPY5sYL2757v7CWAuMKjcOg40Dp43AbZV0M+QYFvMrC3Q2N2XuLsDs4Dbk1G8CMC763fzvf9eSb+OzfnFnbqDnwgkd3C8PbA1ZrkA6FdunaeAN8zsIaAhcEMF/dzFPwOnfdBPbJ/tK/rlZjYWGAuQmZmZYOkisG77QR6YnUfHlg2ZdG9EE/xEAmEPjg8BZrh7BjAQmG1mn9ZkZv2Ao+6+OtGO3X2Su0fcPdKqVavKq1hqhcL9xxgxfRkN60Un+DWprwl+IiclMzgKgQ4xyxlBW6xRwDwAd18MpAMtY14fDDxfrs+M0/QpclYOHC1m+LRlHC0qZcZ9l9Ouaf2wSxKpUpIZHDlAZzPraGZ1iYbA/HLrbAGuBzCzbkSDY1ewXAe4k2B8A8DdtwMHzax/cDbVvcDLSXwPUsscLy5lzKxctuw5yjP39qXr+Y1Pv5FILZO04HD3EuBBYAGwjujZU2vMbLyZ3Ras9hgwxsxWEt2zGBEMegNcA2x19/xyXX8LmAKsBzYAryXrPUjtUlbmfHfeCpZt2svP77yMKy9sefqNRGoh++f3dM0ViUQ8Nzc37DKkCnN3xr+ylunvbuI/BnZjzDWdwi5JJHRmlufukfLtYQ+Oi1QJUxZtZPq7mxj5hSxGX60JfiKfR8Ehtd78ldv48avrGNjzfJ78cnd0MQKRz6fgkFrtHxt28715K8nOas4v7+ylCX4icVBwSK31wScHuX9WHhe0aMDkeyOkp2mCn0g8FBxSK23bf4wR03JoUC+FGfdl06SBJviJxEvBIbXOgWPFjJi+jCNFJcwYmU17TfATSYhu5CS1SlFJKWNn5bJx9xFmjsymW1tN8BNJlIJDao3oBL+VLN24l18P7sWVF2mCn8iZOO2hKjN7yMyanYtiRJLpJ6+u48+rtjPulq4M6lXhRZVFJA7xjHG0AXLMbF5wRz+dryjVztR3NjLlnY2MuDKL+zUrXOSsnDY43P0JonfamwqMAD42s5+Y2YVJrk2kUryyahs/+vNaBvQ4nydv1QQ/kbMV11lVwYUHPwkeJUAz4EUzezqJtYmctSX5e/juCyvpm9mMXw3uRYom+ImctdMOjpvZw0QvX76b6FVpv+/uxcFlzz8GHk9uiSJn5qMdhxg7K5cOzeszZbgm+IlUlnjOqmoOfM3dN8c2unuZmd2anLJEzs72A8cYPm0Z6WkpzLwvm6YN6oZdkkiNEc+hqteAvScXzKxxcEtX3H1dsgoTOVMHjxczcnoOh46XMH3k5WQ0axB2SSI1SjzBMRE4HLN8OGgTqXKKSkq5f1Ye63ceZuI9fejRrknYJYnUOPEEh8XclQ93L0MTB6UKKitzvv/fq1icv4en77iUqzu3CrskkRopnuDIN7PvmFla8HgYKH87V5HQ/efrHzB/5TYeH3AxX+uTEXY5IjVWPMHxAHAlUAgUAP2AsfF0HkwY/NDM1pvZuApezzSzhWa23MxWmdnAmNcuNbPFZrbGzN43s/Sg/e9BnyuCR+t4apGabfq7G3nm7XyG9b+Ab16rKUYiyXTaQ07uvhMYnGjHZpYCTABuJBo4OWY2393Xxqz2BDDP3SeaWXfgVSDLzFKBZ4Fh7r7SzFoAxTHbDXV33URcAHjt/e2Mf2UtN3Vvw1O39dAEP5Eki2ceRzowCugBpJ9sd/f7TrNpNrDe3fODfuYCg4DY4HDg5OVJmwDbguc3AavcfWXwu/ac9p1IrbRs414efmEFvTs05TdDemuCn8g5EM+hqtnA+cDNwFtABnAoju3aA1tjlguCtlhPAfeYWQHRvY2HgvYugJvZAjN7z8zKTzKcHhymevJU184ys7Fmlmtmubt27YqjXKluPt5xiDGzcsloVp+pwy/XBD+RcySe4LjI3Z8Ejrj7TODLRMc5KsMQYIa7ZwADgdnBjPRU4CpgaPDzq2Z2fbDNUHfvCVwdPIZV1LG7T3L3iLtHWrXS2TU1zY6DxxkxPYe0lDrMHJlNs4aa4CdyrsQTHCfHFvab2SVEDynFMyBdCHSIWc4I2mKNAuYBuPtioofCWhLdO3nb3Xe7+1GieyN9gvUKg5+HgOeIHhKTWuTQ8WKGT1vG/qMnmDHycjo01wQ/kXMpnuCYFNyP4wlgPtExiv+MY7scoLOZdTSzukQH2OeXW2cLcD2AmXUjGhy7gAVATzNrEAyUXwusNbNUM2sZrJ8G3AqsjqMWqSFOlJTxwLMnJ/j15ZL2muAncq597uB4cNjooLvvA94G4r6RgbuXmNmDREMgBZjm7mvMbDyQ6+7zgceAyWb2KNGB8hHBZMN9ZvZLouHjwKvu/mczawgsCEIjBXgTmJzge5ZqqqzMefzFlby7fg8//8ZlXNNFhyBFwmAxk8IrXsEs190j56iepIhEIp6bq7N3q7ufvvYBv39rA9+7qQsPXtc57HJEajwzy6vo+z+eQ1Vvmtn3zKyDmTU/+UhCjSKnNGvxJn7/1gbu7pfJt790UdjliNRq8Vxz6q7g57dj2pwEDluJnI3XV3/CD+av4YZubRivCX4ioYtn5njHc1GISEVyN+3l4bnL6dWhKb8d0pvUlLhuWikiSRTPzPF7K2p391mVX47IP63feZhRM3Np1zQ6wa9+XU3wE6kK4jlUdXnM83Sip8++Byg4JGl2HjzO8GnLSEsxZo7Mprkm+IlUGfEcqnoodtnMmgJzk1aR1HqHjhczYnoO+46eYO7Y/mS20AQ/karkTG7IdATQuIckxYmSMr415z0+3HGIKcMjXJrRNOySRKSceMY4/ofoWVQQPX23O8FlQkQqk7sz7qVVLPp4N0/fcSlfuli3WhGpiuLZ4/h5zPMSYLO7FySpHqnFfv7Gh7z0XiGP3tCFOyMdTr+BiIQinuDYAmx39+MAZlbfzLLcfVNSK5NaZfaSzUxYuIEh2R34zvWa4CdSlcVzUvx/A2Uxy6VBm0ileGPNJ/zg5dVc37U1Pxx0iSb4iVRx8QRHqrufOLkQPNe5kVIp8jbv46Hnl9Mzoym/vVsT/ESqg3j+L91lZredXDCzQcDu5JUktcWGXYcZPTOHtk3SmTo8QoO6Z3KSn4ica/H8n/oAMMfMfhcsFwAVziYXidfOQ9EJfnXMmHlfNi3Pqxd2SSISp3gmAG4A+pvZecHy4aRXJTXa4aIS7puRw57D0Ql+F7RoGHZJIpKA0x6qMrOfmFlTdz/s7ofNrJmZ/ehcFCc1T3FpdILfuu2HmDC0N5d10AQ/keomnjGOW9x9/8mF4G6AA5NXktRU7s6/v/Q+b3+0ix/ffgnXdW0TdkkicgbiCY4UM/v0ALSZ1QfiOiBtZgPM7EMzW29m4yp4PdPMFprZcjNbZWYDY1671MwWm9kaM3vfzNKD9r7B8noz+43p3M1q45d/+YgX8wp4+PrODM7ODLscETlD8QTHHOCvZjbKzEYDfwFmnm4jM0sBJgC3EL1MyRAz615utSeAee7eGxgM/FewbSrwLPCAu/cAvggUB9tMBMYAnYPHgDjeg4RsztLN/PZv67kr0oFHbtBtX0Wqs9MGh7v/J/AjoBtwMbAAuCCOvrOB9e6eH8z9mAsMKt890Dh43gTYFjy/CVjl7iuDGva4e6mZtQUau/sSj94sfRZwexy1SIjeXLuDJ/+0mi9e3IoffVUT/ESqu3hnW+0g+iX/DeA6YF0c27QHtsYsFwRtsZ4C7jGzAuBV4OQl3LsAbmYLzOw9M3s8ps/Y62RV1KdUIcu37OPB59/jkvZNmHB3H9I0wU+k2jvl6bhm1gUYEjx2Ay8A5u5fqsTfPwSY4e6/MLMrgNlmdklQ11VEbyJ1lOihsjzgQLwdm9lYYCxAZqaOp4dh4+4jjJqZS+tG6UwbcTkN62mCn0hN8Hl//n1AdO/iVne/yt1/S/Q6VfEqBGIvcZoRtMUaRXCJdndfTPQOgy2J7km87e673f0o0b2RPsH2Gafpk6C/Se4ecfdIq1atEihbKsOuQ0UMn7YMQBP8RGqYzwuOrwHbgYVmNtnMrgcSOTidA3Q2s45mVpfo4Pf8cutsIXorWsysG9Hg2EV0HKWnmTUIBsqvBda6+3bgoJn1D86muhd4OYGa5Bw4UlTCqJk57Dx0nKnDI3RsqQl+IjXJKYPD3f/k7oOBrsBC4BGgtZlNNLObTtexu5cADxINgXVEz55aY2bjY6599RgwxsxWAs8DIzxqH/BLouGzAnjP3f8cbPMtYAqwHtgAvJbwu5akKS4t49vPvcfqwgP8bkgfemc2C7skEalkFj05Kc6VzZoRHSC/y92vT1pVlSwSiXhubm7YZdR47s6//WEV83IL+MlXe3J3P40tiVRnZpbn7pHy7Qmd4uLu+4Kxg2oTGnJuuDu/eOMj5uUW8NB1Fyk0RGowneYiZ83d+elrH/DM2/ncFenAd2/sEnZJIpJECg45K2VlzlP/s4ZZizdzT/9Mxt+mCX4iNZ2CQ85YaZkz7g+r+O+8AsZe04l/v6WrQkOkFlBwyBkpLi3j0RdW8Mqq7Tx8fWceuaGzQkOkllBwSMKOF5fy4HPLeXPdDsbd0pUHrr0w7JJE5BxScEhCjp0oZezsXBZ9vJvxg3pw7xVZYZckIueYgkPidvKWr7mb9vL0HZdyZ6TD6TcSkRpHwSFxOXC0mHunL2N14QF+Nbg3t13WLuySRCQkCg45rd2Hixg2dRkbdh5m4tA+3NTj/LBLEpEQKTjkc+04eJy7Jy+hcP8xJg+PcG0XXWlYpLZTcMgpbd17lKFTlrLncBEzR2bTr1OLsEsSkSpAwSEV2rj7CEMnL+FwUQnPju6nq9yKyKcUHPIZH+04xNApSyktc54f258e7ZqEXZKIVCEKDvkXqwsPMGzqUtJS6jDv/v5c1LpR2CWJSBWj4JBP5W3ex4jpy2icnsZzY/pxQQvduU9EPkvBIQD8Y8NuRs/MpXWjeswZ05/2TeuHXZKIVFEKDmHhhzt5YHYemc0bMGd0P1o3Tg+7JBGpwhK6A2CizGyAmX1oZuvNbFwFr2ea2UIzW25mq8xsYNCeZWbHzGxF8Ph9zDZ/D/o8+VrrZL6Hmu711dsZOyuXi1qfxwv3X6HQEJHTStoeh5mlABOAG4ECIMfM5rv72pjVngDmuftEM+sOvApkBa9tcPdep+h+qLvrJuJn6eUVhXx33kouy2jC9JHZNKmfFnZJIlINJHOPIxtY7+757n4CmAsMKreOA42D502AbUmsR2LMXbaFR15YweVZzZg9qp9CQ0TilszgaA9sjVkuCNpiPQXcY2YFRPc2Hop5rWNwCOstM7u63HbTg8NUT9op7h5kZmPNLNfMcnft2nV276SGmfbORsa99D7XdmnFjJHZNKynoS4RiV9SxzjiMASY4e4ZwEBgtpnVAbYDme7eG/gu8JyZndwzGeruPYGrg8ewijp290nuHnH3SKtWur7SSRMWrmf8K2u5uUcbnhnWl/S0lLBLEpFqJpnBUQjE3rAhI2iLNQqYB+Dui4F0oKW7F7n7nqA9D9gAdAmWC4Ofh4DniB4Sk9Nwd36+4EN+tuBDbu/Vjgl396FeqkJDRBKXzODIATqbWUczqwsMBuaXW2cLcD2AmXUjGhy7zKxVMLiOmXUCOgP5ZpZqZi2D9jTgVmB1Et9DjeDu/PCVdfxu4XoGX96BX9zZi9SUsHc2RaS6StrBbXcvMbMHgQVACjDN3deY2Xgg193nA48Bk83sUaID5SPc3c3sGmC8mRUDZcAD7r7XzBoCC4LQSAHeBCYn6z3UBGVlzn/8aTXPL9vCiCuz+MFXunOKYSERkbiYu4ddQ9JFIhHPza19Z++WlJbx/RdX8cflhXzrixfy/ZsvVmiISNzMLM/dI+XbdTpNDXWipIyH5y7ntdWf8L2buvDgdZ3DLklEaggFRw10vLiUbz6bx8IPd/Hkrd0ZdVXHsEsSkRpEwVHDHCkqYcysXBbn7+EnX+3J3f0ywy5JRGoYBUcNcvB4MSOn57B8yz5+eedlfLV3RtgliUgNpOCoIfYdOcG905bxwScHmXB3H27p2TbskkSkhlJw1AA7Dx1n2JRlbNxzhEnDInypqy4YLCLJo+Co5rYfOMbQyUvZfuA400dczhcuahl2SSJSwyk4qrEte45y95QlHDhazOxR2USymoddkojUAgqOamr9zsMMnbKEopIynhvTn54ZTcIuSURqCQVHNbRu+0HumbIUM2Pu2P50Pb/x6TcSEakkCo5qZsXW/QyftowGdVOYM7ofnVqdF3ZJIlLLKDiqkWUb93LfjByaNUzjudH96dC8QdgliUgtpOCoJhZ9vIsxs3Jp37Q+c0b35/wm6WGXJCK1lIKjGnhz7Q6+Nec9OrVqyLOj+9HyvHphlyQitZiCo4p7ZdU2Hpm7gh7tGjPzvmyaNqgbdkkiUsspOKqwF/MKePzFlfS9oBnTRlxOo/S0sEsSEVFwVFWzl2zmyT+t5urOLdYqd24AAArgSURBVHlmWF8a1NV/KhGpGvRtVAVNfjufH7+6jhu6teZ3d/chPS0l7JJERD5VJ5mdm9kAM/vQzNab2bgKXs80s4VmttzMVpnZwKA9y8yOmdmK4PH7mG36mtn7QZ+/sRp0L1R359dvfsyPX13Hly9ty8R7+io0RKTKSdoeh5mlABOAG4ECIMfM5rv72pjVngDmuftEM+sOvApkBa9tcPdeFXQ9ERgDLA3WHwC8lpx3ce64Oz99/QOeeSufr/fJ4Ok7LiWlTo3JRBGpQZK5x5ENrHf3fHc/AcwFBpVbx4GT18toAmz7vA7NrC3Q2N2XuLsDs4DbK7fsc6+szHlq/hqeeSufe/pn8jOFhohUYckMjvbA1pjlgqAt1lPAPWZWQHTv4aGY1zoGh7DeMrOrY/osOE2fAJjZWDPLNbPcXbt2ncXbSK7SMmfcS6uYuXgzY6/pxA8HXUIdhYaIVGFJHeOIwxBghrtnAAOB2WZWB9gOZLp7b+C7wHNmltCV/Nx9krtH3D3SqlWrSi+8MhSXlvHICyuYl1vAw9d35t9v6UoNGrIRkRoqmWdVFQIdYpYzgrZYo4iOUeDui80sHWjp7juBoqA9z8w2AF2C7WNvpF1Rn9VCUUkpDz63nL+s3cG4W7rywLUXhl2SiEhckrnHkQN0NrOOZlYXGAzML7fOFuB6ADPrBqQDu8ysVTC4jpl1AjoD+e6+HThoZv2Ds6nuBV5O4ntIimMnShk9M5e/rN3B+EE9FBoiUq0kbY/D3UvM7EFgAZACTHP3NWY2Hsh19/nAY8BkM3uU6ED5CHd3M7sGGG9mxUAZ8IC77w26/hYwA6hP9GyqanVG1eGiEu6bkUPOpr08/fVLufPyDqffSESkCrHoyUk1WyQS8dzc3LDL4MDRYoZPX8b7hQf4f3f14rbL2oVdkojIKZlZnrtHyrdr5vg5sudwEcOmLmP9zsNMHNqHm3qcH3ZJIiJnRMFxDuw4eJyhU5ZSsO8ok4dHuLZL1TzLS0QkHgqOJCvYd5ShU5ay+1ARM0dm069Ti7BLEhE5KwqOJNq4+whDJy/hcFEJz47uR+/MZmGXJCJy1hQcSfLRjkMMnbKU0jLn+bH96dGuSdgliYhUCgVHEqwuPMCwqUtJS6nDvPv7c1HrRmGXJCJSacK+5EiNk7d5H0MmL6FB3VTm3X+FQkNEahztcVSixRv2MGpmDq0b1WPOmP60b1o/7JJERCqdgqOS/P3Dndw/O4/M5g2YM7ofrRunh12SiEhSKDgqweurP+Gh59+jS5tGzB7Vj+YN64ZdkohI0ig4ztLLKwr57ryVXJbRhOkjs2lSPy3skkREkkrBcRZeyNnCuJfep1/H5kwdfjkN6+njFJGaT990Z2j6uxv5P/+zli9e3Irf39OX9LSUsEsSETknFBxn4L/+vp6nX/+Qm3u04TdDelMvVaEhIrWHgiMB7s4v//IRv/3bem7v1Y6ff+MyUlM0FUZEahcFR5zcnR/9eR1T39nI4Ms78OOv9iSlju4PLiK1j4IjDmVlzhMvr+a5pVsYcWUWP/hKd6J3rhURqX0UHKdRUlrG4y+u4qXlhXzrixfy/ZsvVmiISK2W1AP0ZjbAzD40s/VmNq6C1zPNbKGZLTezVWY2sILXD5vZ92LaNpnZ+2a2wsySej/YEyVlfGfucl5aXsj3burC4wO6KjREpNZL2h6HmaUAE4AbgQIgx8zmu/vamNWeAOa5+0Qz6w68CmTFvP5L4LUKuv+Su+9OTuVRxaVlPPBsHn/7YCdP3tqdUVd1TOavExGpNpJ5qCobWO/u+QBmNhcYBMQGhwONg+dNgG0nXzCz24GNwJEk1nhKqXWMji0b8uOvXsLQfheEUYKISJWUzOBoD2yNWS4A+pVb5yngDTN7CGgI3ABgZucB/0Z0b+V75bbxYBsHnnH3SRX9cjMbC4wFyMzMTLh4M+PJW7snvJ2ISE0X9iSEIcAMd88ABgKzzawO0UD5f+5+uIJtrnL3PsAtwLfN7JqKOnb3Se4ecfdIq1atklS+iEjtk8w9jkKgQ8xyRtAWaxQwAMDdF5tZOtCS6J7JHWb2NNAUKDOz4+7+O3cvDNbfaWZ/JHpI7O0kvg8REYmRzD2OHKCzmXU0s7rAYGB+uXW2ANcDmFk3IB3Y5e5Xu3uWu2cBvwJ+4u6/M7OGZtYoWL8hcBOwOonvQUREyknaHoe7l5jZg8ACIAWY5u5rzGw8kOvu84HHgMlm9ijRsYsR7u6f020b4I/BKbGpwHPu/nqy3oOIiHyWff73dM0QiUQ8NzepUz5ERGocM8tz90j59rAHx0VEpJpRcIiISEIUHCIikpBaMcZhZruAzWe4eUsgqZc3qWH0eSVGn1di9Hkl5mw/rwvc/TMT4WpFcJwNM8utaHBIKqbPKzH6vBKjzysxyfq8dKhKREQSouAQEZGEKDhOr8KLKMop6fNKjD6vxOjzSkxSPi+NcYiISEK0xyEiIglRcIiISEIUHKdgZtPMbKeZ6eq7cTCzDsH949ea2RozezjsmqoyM0s3s2VmtjL4vP5P2DVVB2aWYmbLzeyVsGup6sxsk5m9b2YrzKxSL9anMY5TCG4QdRiY5e6XhF1PVWdmbYG27v5ecOn7POD2cveYl4BFL/Hc0N0Pm1ka8A7wsLsvCbm0Ks3MvgtEgMbufmvY9VRlZrYJiLh7pU+Y1B7HKbj728DesOuoLtx9u7u/Fzw/BKwjevtgqYBHnbzDZVrw0F9xn8PMMoAvA1PCrqW2U3BIpTOzLKA3sDTcSqq24LDLCmAn8Bd31+f1+X4FPA6UhV1INeHAG2aWZ2ZjK7NjBYdUKjM7D/gD8Ii7Hwy7nqrM3UvdvRfR2ypnm5kOiZ6Cmd0K7HT3vLBrqUaucvc+wC3At4PD75VCwSGVJjhW/wdgjru/FHY91YW77wcWAgPCrqUK+wJwW3Dcfi5wnZk9G25JVZu7FwY/dwJ/BLIrq28Fh1SKYLB3KrDO3X8Zdj1VnZm1MrOmwfP6wI3AB+FWVXW5+7+7e4a7ZwGDgb+5+z0hl1VlmVnD4CQVzKwhcBNQaWeIKjhOwcyeBxYDF5tZgZmNCrumKu4LwDCifwmuCB4Dwy6qCmsLLDSzVUAO0TEOnWIqlaUN8I6ZrQSWAX9299crq3OdjisiIgnRHoeIiCREwSEiIglRcIiISEIUHCIikhAFh4iIJETBIVIJzKw05jTkFWY2rhL7ztJVmqUqSQ27AJEa4lhw+RCRGk97HCJJFNwT4engvgjLzOyioD3LzP5mZqvM7K9mlhm0tzGzPwb36VhpZlcGXaWY2eTg3h1vBLPNRUKh4BCpHPXLHaq6K+a1A+7eE/gd0Su8AvwWmOnulwJzgN8E7b8B3nL3y4A+wJqgvTMwwd17APuBryf5/YickmaOi1QCMzvs7udV0L4JuM7d84OLQH7i7i3MbDfRG18VB+3b3b2lme0CMty9KKaPLKKXJOkcLP8bkObuP0r+OxP5LO1xiCSfn+J5Iopinpei8UkJkYJDJPnuivm5OHj+D6JXeQUYCiwKnv8V+CZ8eqOnJueqSJF46a8WkcpRP7ib30mvu/vJU3KbBVfBLQKGBG0PAdPN7PvALmBk0P4wMCm4GnMp0RDZnvTqRRKgMQ6RJArGOCLuvjvsWkQqiw5ViYhIQrTHISIiCdEeh4iIJETBISIiCVFwiIhIQhQcIiKSEAWHiIgk5P8DXMGABnI0gcIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = build_ft_model()\n",
    "model.summary()\n",
    "train_model(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fine-tuning achieves better accuracy, compared to the feature-engineering model. The most likely reason is that the feature-engineering model is constrained to update only the weights of the last hidden layer."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
